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FORWARD 


The Survey and Audit Working Group was assembled to establish, train personnel, 
and implement a process oriented Survey and Audit Program that monitors and 
measures capability, compliance, effectiveness, and corrective action for safety, 
reliability, maintainability, and quality -- both in "doing" (engineering, operation, 
maintenance) and "assuring" (SRM&QA) activities. The Survey and Audit Working 
Group consists of representatives from many diverse organizations who each have 
various degrees of Survey and Audit responsibilities. This Survey and Audit 
Program Plan was written by representatives from each of the organizations bsted 
below. These representatives, by sig nin g below, give their concurrence to the Plan 
and its requirement to provide an improving Survey and Audit Program at the John 
F. Kennedy Space Center. 
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KSC SURVEY AND AUDIT PROGRAM 
SECTION I 


INTRODUCTION 

1.1 PURPOSE AND APPLICABILITY 

This document establishes and defines the program applicable to the John F. 
Kennedy Space Center (KSC) for conducting surveys and audits of Safety, 
Reliability, Maintainability, and Quality (SRM&Q) activities, and SRM&Q 
Assurance activities. The program provides standardized terminology, 
uniformity of survey and audit operations, compliance and process 
assessments, and indicators which monitor and measure capability, 
compliance, effectiveness, and corrective actions. The program is applicable 
to KSC civil service and contractors and describes the assessment process, 
identifies metrics, specifies required skills training, and provides guidance 
for implementation. 

1.2 SCOPE 

This program applies to all primary contractor and civil service organizations 
responsible for implementing Safety, Reliability, Maintainability, and 
Quality requirements at the Kennedy Space Center. The program has 
implications for off-site vendors and suppliers. 

1.2.1 On-Site Elements/Subordinate Organizations 

All organizations with Safety, Reliability, Maintainability , and Quality 
Assurance responsibilities or having processes which impact the 
success of KSC mission goals and objectives shall be subject to audits, 
surveys, and assessments as defined herein. 

1.2.2 Off-Site Vendors/Suppliers 

All vendors and suppliers with Safety, Reliability, Maintainability, 
and Quality Assurance responsibilities written into their contract shall 
be subject to audits, surveys, and assessments as defined herein. 
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1.3 APPLICABLE DOCUMENTS 


KHB 5310.1, Reliability, Maintainability and Quality Assurance 
Handbook 

KMT 1710.1, Safety, Reliability, Maintainability and Quality Assurance 
Programs 

KPD 8710.1, KSC/Safety, Reliability, Maintainability and Quality 
Assurance Programs 

K-RQ-0001.3.1, NASA KSC Assurance Sampling and Measurements 
Manual 

NHB 1700.1(VI-B), NASA Safety Policy and Requirements Document 

NHB 5300.4 (ID-2), Safety, Reliability, Maintainability and Quality 
Provisions for the Space Shuttle Program 

NHB 5300.4 (IB), Quality Program Provisions for Aeronautical and Space 
System Contractors 

NSTS 22579, Space Shuttle Program SR&QA Survey Program Plan 


1.4 DEFINITIONS 

Audit A systematic, independent, official, examination 

and verification of records, and other objective 
evidence, of work performed, of the process, or of 
the process requirements to determine compliance 
to requirements, to assess the effectiveness of 
implementation, and to identify potential 
improvements. 

Auditor/Surveyor An individual qualified through training and 

experience to perform surveys and audits. As used 
herein, the term auditor applies to both auditor 
and surveyor. 

Auditor, Certified An auditor certified by management as having the 

training identified in Section IV, paragraph 4.2.1, 
of this plan. 
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Auditor, Senior 


Findings 


Survey 


An auditor certified by management as having the 
training identified in Section IV, paragraphs 4.2.1 
and 4.2.2. 

Documented results of investigations and 
evaluations which are based on substantiating 
evidence. A Finding may be: 

a. A "Nonconformance" identifying a deviation 
from requirements. 

b. An "Observation" identifying a condition or 
practice that should be corrected to improve 
a process. 

c. A "Verification" indicating compliance to a 
requirement. 

d. A "Commendation" recognizing outstanding 
performance. 

An independent, official, comprehensive evaluation 
or assessment of capabilities to ensure that 
programmatic systems are adequately documented, 
effectively implemented, and suitable for achieving 
requirements and desired quality objectives. 
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KSC SURVEY AND AUDIT PROGRAM 
SECTION II 


BACKGROUND 

The critical nature of launch preparations at the Kennedy Space Center requires a 
high degree of checks and balances to assure work is properly accomplished. One of 
these checks and balances is a proactive government and contractor survey and 
audit program that provides management with an evaluation of Safety, Reliability, 
Maintainability, and Quality Assurance (SRM&QA) Programs. In the past, the 
major thrust of government and contractor surveys and audits was an evaluation of 
compliance to requirements. However, the KSC Safety, Reliability and Quality 
Assurance Directorate (RQ) recognized the inherent limitations in relying solely on 
compliance oriented surveys and audits. The survey and audit program should 
include assessment elements that can monitor and measure capability, 
effectiveness, and corrective action. Therefore, in 1991, RQ management made a 
presentation at a NASA Headquarters (Code Q) chaired R&QA Directors meeting 
emphasizing a process oriented survey and audit program for the Centers. NASA 
Headquarters expressed interest in the concept and KSC has been the lead Center 
•in developing this improved approach. 

During the implementation of the Structured Surveillance Program, a requirement 
was defined to provide assessments of the effectiveness of Structured Surveillance 
Program elements. Therefore, a "Structured Surveillance Survey and Audit 
Working Group" was established, and the first meeting was held on June 21, 1992. 
The working group was comprised of KSC civil service and contractor SRM&QA 
representatives involved in conducting off-site and on-site surveys and audits. 

After analyzing the various presentations involving the current methodology 
employed by the Civil Service and contractor organizations in performing surveys 
and audits, it became apparent that there were more differences than similarities. 
To effectively support the Structured Surveillance Program, a KSC Survey and 
Audit Program, with some degree of standardization, would be required so that 
statistical data would be valid across organizational interfaces. In addition, since 
compliance audits are limited along organizational lines, a broader "process 
oriented" survey and audit methodology would be needed. It was recognized by the 
working group that improvements in the survey and audit methodology would have 
a beneficial effect on assessing engineering, operations, maintenance, and 
SRM&QA programs. 
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Therefore, the members agreed that the working group's original objectives should 
be expanded to develop a KSC Survey and Audit Program (including process 
oriented audits) for both on-site and off-site assessments. The title for the working 
group was changed to the "KSC Survey and Audit Working Group." 
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KSC SURVEY AND AUDIT PROGRAM 
SECTION III 


APPROACH 
3.1 GENERAL 

The survey and audit function is a service function which intends to facilitate 

the attainment of Safety, Reliability, Maintainability and Quality (SRM&Q) 

by providing leadership and management of the survey and audit process. 

The function encompasses on the following activities: 

A. The attainment of SRM&Q within peer organizations through surveys 
and audits. For example, the Payloads Management and Operations, 
Shuttle Management and Operations, and Installation Management 
and Operations Directorates, are responsible for SRM&Q self- 
assessments and for requesting surveys and audits by the Safety, 
Reliability and Quality Assurance Directorate. The Safety, Reliability 
and Quality Assurance Directorate is responsible for surveys and 
audits of SRM&Q on a periodic basis in the absence of requested 
services sufficient to provide adequate assessments. The intent is to 
independently monitor and measure processes by involving SRM&QA 
in project management, engineering, manufacturing, operations, and 
maintenance. 

B. The attainment of SRM&Q within contractor organizations 
subordinate to peer organizations through surveys and audits. For 
example, the Safety, Reliability and Quality Assurance Directorate is 
responsible for monitoring, measuring, and evaluating SRM&Q in 
engineering, operations, and maintenance activities of the Payload 
Ground Operations Contract (PGOC), Shuttle Processing Contract 
(SPC), Base Operations Contract (BOC), etc. 

C. The attainment of SRM&Q Assurance (SRM&QA) in contractor 
organizations subordinate to peer organizations through surveys and 
audits. As an example, this element of the activity would include the 
civil service monitoring and measuring of PGOC, SPC, and BOC 
SRM&QA and all of the KSC off-site survey and audit activities. 

D. The attainment of SRM&Q Assurance (SRM&QA) through surveys 
and audits of SRM&QA processes and SRM&QA self-assessment 
activities. 
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E. Special surveys and audits of processes, civil service organizations and 
contractor organizations when management decides such action is 
warranted due to results of routine monitoring or other significant 
events. 

3.2 CONCEPT 

The KSC Survey and Audit Program evolved from a program focused 
primarily on determining conformance to requirements to a program which 
evaluates the effectiveness of processes. The performance of surveys and 
audits is accomplished by a team composed of a leader (Senior Auditor) 
trained in planning, organizing, scheduling, closing, and reporting on survey 
and audit results; and subject matter experts (Auditors) trained in 
interviewing, investigating, examining records, analyzing; and documenting 
findings. On an as needed basis, this team is augmented by subject matter 
experts not trained as auditors. In these cases, the individuals are under the 
guidance of an auditor. 

3.2.1 Compliance Oriented, Vertical Assessment 

The previous SRM&QA survey and audit function had provided 
assurance of compliance to requirements and was "compliance" 
oriented. Requirements are imposed in a contract and the contractor 
maintains an SRM&QA organization in its management structure 
through which the responsibility for assuring compliance to 
requirements is passed down. Surveys and audits were generally 
performed on the SRM&QA organization to verify that contract 
requirements were passed down through procedures. The survey and 
audit method was a "vertical" assessment of the SRM&QA 
organizations’ performance and compliance to requirements. 

The present evaluation of compliance does not consider the relevancy 
of requirements. Today's continuous improvement environment 
challenges people to create innovative enhancements. A survey and 
audit function that strictly forces compliance to requirements is 
counterproductive to continuous improvement. The survey and audit 
function has been improved to include an assessment of requirements 
to ensure that they are adequate, necessary, and achieve the desired 
goals and objectives. Compliance to a requirement must add value to 
the product or system. 
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3.2.2 Process Oriented, Horizontal Assessment 

The improved survey and audit methodology indudes the evaluation of 
process conformance and effectiveness. Processes are the sequences of 
events and tasks leading to an end product and are generally 
documented in procedures that detail the performance responsibilities 
of all organizations involved. Since the "process" oriented survey and 
audit crosses contractor and organizational boundaries, it is considered 
a "horizontal" assessment. 

By evaluating processes, all organizations are induded in the 
examination and analysis. Process analysis is multifunctional 
(engineering, operations, SRM&QA) in order to properly assess 
effectiveness and implement improvements. The "process" oriented 
survey and audit function seeks solutions to problems and their root 
causes, thereby increasing the occurrence of first-time quality. 

3.3 PROGRAM DESCRIPTION 

The KSC Survey and Audit Program provides uniformity between the 
various organizations performing survey and audit activities at KSC. In 
addition to improving these activities by emphasizing process evaluation and 
assuring value added compliance to requirements, the program fosters 
continuous improvement through the use of measurement indicators (see 
Section V), provides for competent execution through skills training (see 
Section IV), and establishes a centralized scheduling and reporting effort to 
allow enhanced coordination and feedback between organizations performing 
surveys and audits at KSC. 

3.3.1 KSC Organizations Performing Surveys/Audits 

The Safety, Reliability and Quality Assurance Directorate (RQ) has 
the primary responsibility for providing SRM&QA surveys and audits 
at KSC. The function is performed by an organization within the 
Mission Assurance Directorate (RM), the Procurement Assurance and 
Audit Integration Branch accomplishing off-site and on-site surveys 
and audits. In addition, KSC Operational Directorates (TM, CM, and 
IM) have prime contractors that are responsible for performing on-site 
and off-site surveys and audits. 

Diagrams 1A and IB show the responsibilities and functions of the two 
levels of organizations at KSC involved in survey and audit activities. 
The Survey and Audit Branch provides assessments for compliance and 
effectiveness of NASA KSC Operational Organizations (including the 
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RQ Directorate) and KSC on-site contractors, and assessments of 
capability, compliance, and effectiveness of off-site contractor and 
vendor operations. KSC contractor organizations are responsible for 
surveys and audits that provide compliance and assessments of the 
effectiveness of internal organizations and on-site subcontractors. In 
addition, KSC contractor organizations are responsible for surveys and 
audits that provide capabilities, compliance and effectiveness 
assessments of off-site subcontractor and vendor operations. 

3.3.2 Survey and Audit Process 

Definitions for a survey and an audit are provided in paragraph 1.4. 
However, the distinguishing difference is not always clear because of 
the many different functions being performed that are titled surveys 
and audits. Regardless of the name of the survey or audit, the end 
result of the function being performed is to assess capability, 
compliance, effectiveness, or a combination thereof. Diagram 2 
presents the three basic phases common to all surveys and audits: 
planning, performing, and concluding survey and audit activities. It 
also depicts the continuous improvement aspect by returning to the 
planning phase in a closed loop. 

A. During the planning phase, survey and audit activities are 
scheduled, the organizations to be surveyed and audited are 
officially notified, the conduct of the survey and audit is 
organized (resulting in a plan of auditor and team assignments 
and areas to be investigated), and previous audit results, 
contracts, plans, issuances specifying requirements, and 
procedures describing processes are reviewed. In addition, the 
auditors prepare checklists and questions to be asked during 
interviews and the auditors attend entrance briefings to 
determine points-of-contact, arrange meetings, and determine 
facility locations. 

B. During the performance phase, the auditors meet with points-of 
contact and local area management, interview personnel, and 
examine records in accordance with the checklists. Processes 
are reviewed and analyzed for potential improvements. The 
auditors also attend team meetings with personnel from the 
organization being evaluated to discuss issues. The auditors 
investigate problems, noncompliances, and process 
enhancement opportunities and attempt to resolve issues with 
area management. If unsuccessful, they attempt to resolve 
issues with the functional managers within the organization 
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Diagram 2 





being evaluated. If still unsuccessful or the value added of a 
requirement is in question, they attempt to resolve issues with 
cognizant SR&QA management. Auditors will have unimpeded 
access to the level of management necessary to resolve an issue. 
The results are findings and recommendations for improvement. 

C. During the concluding phase, the auditors evaluate the results 
of investigations and, if determined to be value-added, 
document the findings (nonconformances) with 
recommendations for corrective action, including those 
corrective actions sig nifi cant, enough to require statusing until 
completed. Opportunities for improvement resulting from 
process evaluations are also developed. Debriefings to upper 
level management are made (when appropriate) and the final 
report is prepared and distributed. When the audited 
organization submits corrective action plans, responses, or 
status, the auditors evaluate the information, status or close the 
items, and provide survey and audit closure and followup, as 
necessary. The survey or audit will normally be closed with the 
acceptance of a corrective action plan. 

3.4 PROGRAM REQUIREMENTS 

Paragraph 1.3, APPLICABLE DOCUMENTS, lists issuances that contain the 
basic NASA and contractor requirements for performing surveys and audits. 
The following paragraphs supplement these general requirements in order to 
ensure uniform implementation of the KSC Survey and Audit Program. 

3.4.1 Survey and Audit Plans and Procdures 

NASA and contractor organizations at KSC performing survey and 
audit functions shall have a Survey and Audit Program Plan using 
Appendix A, SURVEY AND AUDIT PLAN GUIDE, and/or procedures 
sufficient to implement the program. The plan shall define 
organizational responsibilities as related to survey and audit 
activities, the approach to be used in implementing the program, a 
description of the applicable on-site and off-site activities, auditor 
training and certification, and the intended use of measurements and 
trending. 
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3.4.2 Process Evaluations 

Process evaluations represent the core enhancement provided by the 
survey and audit program. A process evaluation consists of a study 
and examination of the following: 

A. The sequential flow of data and material required to produce a 
service or product. 

B. The tasks associated with transforming the data and material 
into useful information or the constituents of the final product. 

C. The functions of personnel involved in the events occurring 
during the process. 

D. The interrelationships of other processes. 

E. The requirements and control aspects of the process. 

F. The problems, noncompliance issues, and instabilities of the 
process. 

G. The measurements that define the efficiency, effectiveness and 
acceptability of the process, and process results. 

The information obtained during the study and examination is 
analyzed using techniques, such as, flow diagrams, Pareto charts, 
cause and effect diagrams, trends of measurement data and problems, 
and any other tools that will simplify and provide basic understanding. 
The analysis should be sufficient to permit visibility of the root causes 
of problems, recognition of potential improvements, and the 
determination that the process yields the desired results with 
minim um effort and error. 

Survey and Audit Plans and/or procedures shall delineate process 
evaluations and the methods utilized to provide process improvements 
and promote first time quality. 

3.4.3 Compliance Evaluations 

Compliance evaluations are an integral part of the survey and audit 
program. Determining compliance requires the existence of 
substantiating evidence. Compliance evaluations consist of a study 
and interpretation of requirements by the auditor, the selection of 
criteria that the auditor believes will provide the knowledge necessary 


14 



to be able to assert compliance (checklist), interviews with audited 
personnel, and an examination of records that may be performed by 
statistical sampling. 

The existence of a nonconformance should initiate further 
investigation. A nonconformance that goes undetected until an audit 
should inspire questions about the necessity of the requirement, the 
priority and criticality of the requirement, the resources needed for 
compliance, and the value that is added to the system or process by 
requiring compliance. 

Survey and Audit Plans and/or procedures shall delineate compliance 
evaluations and the methods to be utilized in preventing costly and 
unnecessary reporting of unwarranted nonconformances. 

3.4.4 Capabilities Evaluations 

Capabilities evaluations routinely constitute the part of the survey and 
audit program that currently addresses off-site contractors, vendors, 
and suppliers; however, the on-site survey and audit program may 
perform a capability evaluation for ongoing or new contractors, as 
needed. A capabilities evaluation is an assessment of personnel, 
equipment, facilities, management structure, quality, reliability and 
maintainability programs and systems, documentation, procedures, 
material, product, and any other items necessary to perform or produce 
in accordance with contract requirements. The evaluation results in 
the determination that a contractor has or does not have the resources 
and ability to ensure a reasonable probability of successfully 
completing the contract. 

Survey and Audit Plans and/or procedures shall delineate capabilities 
evaluations and the methods utilized to ensure adequate assessments 
and accurate determinations. 

3.4.5 Scheduling and Results Reporting 

The KSC Survey and Audit Program provides sufficient uniformity to 
permit consolidation of survey and audit schedules and reported 
results. NASA will be responsible for consolidating this information 
and providing it to all organizations performing survey and audit 
activities at KSC. This will provide greater program visibility and 
encourage coordination resulting in increased efficiency. 

Contractor Survey and Audit Plans and/or procedures shall provide for 
the reporting of survey and audit scheduling and summarized results 
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with distribution to indude NASA. The NASA plan and procedures 
shall provide for consolidation of survey and audit scheduling and 
summarized results with distribution to indude all KSC organizations 
performing survey and audit activities. 

3.4.6 Training and Certification 

Training and certification requirements are identified in Section IV, 
TRAINING AND CERTIFICATION. Survey and Audit Plans and/or 
procedures shall indicate auditor training requirements. 

3.4.7 Evaluation Procedure and Measurements 

The evaluation procedure and measurement requirements are 
identified in Section V, EVALUATION PROCEDURE AND 
MEASUREMENTS. Survey and Audit Plans and/or procedures shall 
identify appropriate measurements and provide methods and 
procedures for trending, including response actions based on 
measurement results. 
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KSC SURVEY AND AUDIT PROGRAM 
SECTION IV 


TRAINING AND CERTIFICATION 

4.1 GENERAL 

The workforce involved in the survey and audit function needs to have a 
common understanding of the way Kennedy Space Center (KSC) performs 
this highly integrated surveillance of SRM&Q and SRM&Q Assurance 
functions. To ensure that this common understanding is achieved, specific 
training is necessary. An evaluation of the degree of learning for each 
training activity must be made as a condition of course completion. 

The survey and audit workforce is composed of a cadre of experts in the 
survey and audit disciplines (Senior Auditors) who are to plan, organize, 
implement, report, and status survey and audit activities, and subject matter 
experts (Auditors) who perform survey and audit functions. The inclusion of 
subject matter experts as ad hoc members of the survey and audit workforce 
dictates the need for identifying two levels of training. These training 
requirements are not intended to exclude technical experts, assistants, or 
auditors in training from participating in a survey or audit, provided they 
are given guidance by an auditor trained in the manner described herein. 

KSC organizations performing survey and audit functions will identify 
auditor training requirements, courses, methods for providing training, a 
status and record keeping system, and provisions for auditor certification in 
their Survey and Audit Plan and/or procedures. 

4.2 TRAINING REQUIREMENTS 

The training requirements for Auditors and Senior Auditors are described in 
the following paragraphs. 

4.2.1 Auditor 

An auditor must be trained in survey and audit preparation 
techniques, such as, documentation and requirement review, 
checklists, and analysis of previous survey and audit measurement 
results. Training will include techniques for interviewing, records 
examination, investigating, process analysis, and effectiveness 
measurement interpretation. Training will also provide for post- 
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survey and audit activities, such as, documenting results and 
recommendations, evaluating responses, and closing documented 
items. 

4.2.2 Senior Auditor 

In addition to the training for an auditor, a Senior Auditor must be 
trained in planning and organizing surveys and audits, selecting team 
members, scheduling and chairing briefings and meetings, and 
resolving conflicts. In addition, Senior Auditor training will provide 
for the preparation of survey and audit reports, debriefing 
management, and survey and audit closure. 

4.3 TRAINING CRITERIA 

The training course content described below represents the minim um criteria 
suitable for meeting training needs defined by this program initiative. This 
training is in addition to any training required to perform the technical or 
administrative part of an individual's task. Should any of these courses 
overlap other required training, the individual may elect to be evaluated and 
not take the entire course. 

Survey and Audit Preparations: Training associated with presurvey and 
audit activities including analysis of previous survey and audit measurement 
results; reviewing requirement documentation; conducting initial interviews 
with the personnel being audited; determining cognizant points-of-contact; 
and checklist preparation. 

Survey and Audit Techniques: Training associated with the performance 
of survey and audit activities including interviewing personnel, records 
examination, investigating problem causes, analyzing processes, 
measurement techniques, and evaluating the effectiveness of procedures and 
requirements in obtaining desired objectives. 

Postsurvey and Audit Activities: Training associated with postsurvey 
and audit activities including documenting results, assessing performance 
measures, making recommendations for process enhancement or corrective 
action, evaluating responses, and closing documented items. 

On-The-Job-Training: Training associated with the actual performance of 
the activities described above. 
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Organizing and Leading a Survey or Audit (Senior Auditor 
Requirement): Training associated with planning, orga ni z in g and 
scheduling surveys and audits including team member selection, chairing 
briefings and meetings, and resolving conflicts. 

Survey and Audit Report Preparation (Senior Auditor Requirement): 
Training associated with performing postaudit meetings, preparation of 
executive summaries, survey and audit reports, debriefing management, and 
closing surveys and audits. 

4.4 ACCREDITATION 

Candidates having completed the required training, which includes an 
assessment of learning, will be evaluated and certified by their management. 
Management should use relevant experience and past performance as 
demonstrating required skills. 

4.6 MAINTENANCE OF ACCREDITATION 

Each organization shall address maintenance of accreditation and 
suspension requirements in their Survey and Audit Plan and/or procedures. 
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KSC SURVEY AND AUDIT PROGRAM 
SECTION V 


EVALUATION PROCEDURE AND MEASUREMENTS 

5.1 INTRODUCTION 

An evaluation procedure and measurement system are required for the 
survey and audit program. Each will provide information on the 
performance of the organization or process being surveyed or audited 
(auditee) and the organization performing the survey or audit. The 
performance information will be used to decide on appropriate, meaningful 
recommendations, and assess the impact of previous changes or corrective 
actions. The evaluation procedure is a structured method of assessing each 
finding. The measurement system provides metrics which will be used for 
continuous improvement by both the auditee and the auditing organization. 
The attributes of performance that must be evaluated are capability, 
compliance, effectiveness, and corrective action. 

Contractor and civil service Survey and Audit Plans and procedures shall 
describe an evaluation procedure and measurement system to evaluate 
performance attributes including how data is collected, interpreted, utilized, 
and reported. The data must provide the means to indicate trends in the 
auditee's SRM&Q functions and SRM&QA programs. 

6.2 SURVEYS 

5.2.1 Preaward Surveys 

Preaward surveys are performed to evaluate a contractor’s, 
subcontractor's, or vendor's ability to meet requirements prior to 
contract award. 

The preaward survey does not determine compliance since there is no 
contractual relationship. Therefore, the performance measure is 
limited to capability. Measurement data for preaward surveys is in 
the form of "acceptable" (capable) or "unacceptable" (not presently 
capable). Alternatives to this measurement approach may be proposed 
for evaluation in each organization's Survey and Audit Plan and/or 
procedures. 
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5.2.2 Postaward Surveys 

Postaward surveys are performed to evaluate a contractor's, 
subcontractor's, or vendor's compliance, and may include process 
evaluations. Performance measurements will be accumulated 
consistent with SRM&Q functions and SRM&QA requirements. The 
auditor is required to make value judgments based upon many factors 
in order to determine the significance of survey findings. The 
evaluation procedure described in paragraphs 5.4 and 5.5 provides the 
data necessary for assessing compliance and effectiveness during 
postaward surveys. Tailoring or departure from this procedure for 
off-site postaward surveys may be proposed for evaluation in each 
organization’s Survey and Audit Plan and/or procedures. 

5.3 AUDITS 

5.3.1 Compliance Audits 

Compliance audits are assessments, usually involving a checklist, of a 
contractor’s, subcontractor's, or civil service organization's compliance 
to SRM&Q and SRM&QA requirements. The auditor is required to 
make value judgments based upon many factors in order to determine 
the significance of audit findings. The evaluation procedure described 
in paragraphs 5.4 and 5.5 provides the data necessary for assessing 
compliance audits. 

5.3.2 Process Audits 

A process audit is an assessment of procedures, practices, and systems 
to evaluate the effectiveness of implementation and to identify 
potential improvements. The auditor is required to make value 
judgments based on many factors in order to determine the 
significance of findings. The evaluation procedure described in 
paragraphs 5.4 and 5.5 provides the data necessary for assessing 
process audits. 

5.4 CONCEPT 

Since compliance issues may arise during a process audit and 
recommendations to improve a process may be made during a compliance 
audit, an evaluation procedure and measurement system that can 
accommodate both compliance and process audits is required. The following 
paragraphs describe a uniform method for evaluating each finding and a 
basic measurement system that is acceptable for the purpose of providing the 
measurement results required for the KSC Survey and Audit Program. 
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5.4.1 Measurement Introduction 

The results from a survey or audit are findings of: (1) 

Nonconformances requiring corrective action; (2) Observations with 
recommendations for improvement; (3) Verifications of compliance to 
requirements; and (4) Commendations recognizing outstanding 
performance. The input to the measurement system is the data 
collected through the evaluation procedure. The output of the 
measurement system must be information that can be used to judge 
success. To accomplish this, the information must quantify 
compliance, effectiveness, and corrective action. The function of the 
measurement system is to translate findings into measures of the 
attributes of performance. 

5.4.2 Common Characteristics of Findings 

The first step in the evaluation is to identify common characteristics of 
findings that provide information about the significance of a problem 
or enhancement as related to what is affected and to what degree, the 
magnitude as related to how often it occurs, and the impact as related 
to the effect or result of taking corrective action. The categorization of 
information about each finding can then be defined as Importance, 
Character, Frequency, and Impact. 

5.4.3 Establishing Levels of Significance 

The next step is to provide for levels of the finding characteristics 
identified above. "Importance" ranges from critical to minor and is defined 
as: (1) Critical - effecting the safety of personnel or the condition of flight 
hardware; (2) Mission - having the potential to effect mission success; (3) 
Economic - regarding waste or cost; and (4) Minor - indicative of attention 
to detail and first time quality. "Character" ranges from systemic to 
random and is defined as: (1) Systemic - inherent in systems that define 
and implement SRM&Q functions or SRM&QA programs; (2) 
Organizational - limited to function groups; (3) Area * relating to a 
particular location or facility; and (4) Random - meaning isolated and 
probably unrelated. "Frequency" ranges from all the time to infrequently 
and is defined as: (1) Always, (2) Usually, (3) Sometimes, and (4) Rarely. 
"Impact" is defined as: (1) Essential to maintaining, enhancing, and 
ensuring safety; (2) Required for improving first time quality; (3) Desirable 
for improving product quality and outgoing quality; and (4) Undesirable 
addition of cost. These levels are depicted in Diagram 3. The Importance 


22 




Levels of FtadinM Characteristics 



23 






of a finding is summarized with all other findings of a similar level of 
Importance. The same is accomplished for Character, Frequency, and 
Impact of the findings. The total of each column should be equal. 

5.4.4 Data Organization 

The last step is to recognize that the performance attributes are 
associated with different types of findin gs. This requires that levels of 
the characteristics for Nonconformances, Verifications, Observations, 
and Commendations be accumulated separately. 

5.4.5 Acceptability of Recommendations 

When documenting Observations, the auditing organization provides 
recommendations for improvements. The audited organization 
responds to the acceptability of the recommendations in a corrective 
action plan. This information is used in measuring the effectiveness of 
the auditing organization. 

6.4.6 Corrective Action Performance 

Since audits may be closed based on the auditee's corrective action 
plan, a measurement is necessary to assess the auditee's progress. 

Data on the number of commitments achieved out of the total due will 
be used for this measurement. This measurement may be performed 
at a major milestone identified in the plan or at the estimated 
completion date. 

5.5 EVALUATION PROCEDURE 

This procedure and the measurement system outlined in paragraph 5.6 will 
provide information on the performance of the organization or process being 
surveyed or audited (auditee) and the organization conducting the survey or 
audit. Analysis of the metrics will allow for continuous improvement by both 
the auditee and the auditing organization. The procedure and measurement 
sequence is identified in Diagram 4 and delineated in the paragraphs below. 

5.5.1 Prepare 

The evaluation procedure must begin with the development of 
checklists and a review of documentation describing the process being 
audited. The auditor must use this information and measurement 
results from previous surveys and audits to ensure that the eventual 
results of the survey or audit contain a sufficient number of significant 
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items to provide a comprehensive and credible examination that will 
yield adequate data for measurement indicators. To avoid the 
anomalies associated with small sample sizes, the number of 
significant items in the audit plan should be greater than 35. Audits 
that have fewer than 35 must contain information necessary to justify 
the deviation. When preparing for a process audit, the inputs and 
outputs of each step in the process must be determined and evaluated. 

5.5.2 Perform 

The auditor should endeavor to obtain necessary information in the 
least intrusive and disruptive manner possible. Complete information 
is necessary to ensure the accuracy of the metrics and this requires 
auditee cooperation. The auditor should not rely solely on a checklist 
or adhere to predetermined items when more potentially valuable or 
significant information arises (unless continuity is important). In 
addition, the auditor should always provide the auditee with an 
opportunity to identify successes and implemented improvements or 
enhancements which may be documented as Commendations. 

In addition, sampling should be utilized whenever possible to simplify 
the audit process and provide meaningful data for measurement. 
Inspecting a large number of items to produce a small number of 
nonconformances could skew the measurement data. 

5.5.3 Document 

The auditor should document findings as indicated in Diagram 5. 
Outstanding process effectiveness is documented as a Commendation. 
Areas needing improvement are documented as Observations. Process 
ineffectiveness determined to be the result of a lack of adherence to 
requirements is a noncompliance issue and should be documented as a 
Nonconformance. 

Outstanding methods of compliance should be identified in a 
Commendation. Verifications are used to record acceptable (Good) 
compliance and Nonconformances are used to record unacceptable 
(Bad) noncompliance. 

Nonconformances should not be used to document a remedial action 
necessary to correct a discrepancy. Discrepancies do not provide 
adequate or useful information for trending purposes. 
Nonconformances must identify a correctable problem and the cause. 
In addition, a judgment of the value added by a requirement should be 
made. Compliance to a requirement that does not add value is "Not 
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Good" and should be identified in an Observation. Further, 
noncompliance with a requirement that does not add value is "Not 
Bad" and should also be identified in an Observation. 

5.5.4 Validate 

The auditor should validate requirements by ensuring that they are 
necessary to achieve the desired result of the process. Questions 
concerning the accuracy or necessity of requirements should be 
resolved through discussions with local area management, auditee 
functional management, or cognizant SR&QA management. The 
results of the discussions may require further investigation. Auditing 
organization metrics described in paragraph 5.6.6 will reflect the 
success of this task. 

5.5.5 Evaluate Significance 

The levels of significance are ranked from top to bottom in each of the 
four characteristics (Diagram 3), with the highest significance at the 
top. When evaluating the significance of a finding (Nonconformance, 
Verification, Observation or Commendation), use good judgment and 
all available information that pertains to a reasonable probability of 
occurrence or effect, not remote possibility. Start with the first 
characteristic (Importance) and the most significant level (Critical) 
and decide if the importance of the finding is considered critical. If not 
Critical, refer to the next lower level (Mission) and decide if the 
importance is considered mission related. If not Mission, continue to 
move down, using the lowest level whenever the higher levels are not 
appropriate. Next move to the second characteristic (Character) and 
perform a "top-down" analysis. Continue the process for all four 
characteristics. 

The following paragraphs are intended to provide a brief description of 
the evaluation of findings. 

A. Importance - When determining the Importance of a finding, if 
it isn't obviously Critical, Mission related, or it doesn't involve 
significant Economic value, it's Minor. This raises the most 
important findings to a high level of visibility. Critical and 
Mission related findings are candidates for immediate action. 
Findings of an Economic value should be judged to be more 
significant than the average Minor. 

B. Character - When determining the finding's Character, Systemic 
should be used to identify problems with contract requirements, 
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quality plans, procedures, and any directive that causes the 
wrong thing to be done. Discipline problems are generally 
Organizational in Character, even if they appear randomly. 
Findings that record facility conditions, or are only meaningful 
when considering the location, should be summed as Area. If 
the finding's Character cannot be determined to be Systemic, 
Organization, or Area, it should be summed as Random. 

C. Frequency - The Frequency of a condition or occurrence reported 
in a finding should be determined as Always, Usually, 
Sometimes, or Rarely. For example, if the finding identifies a 
safety hazard that is part of the building, it is Always present. 

If the hazard exists when a particular machine is in use or only 
during a specific operation, it is present Sometimes. 

D. Impact - The value of Impact should reflect the effect of taking 
corrective action or implementing a recommendation. Problems 
that effect Safety have the highest priority. 

6.6 MEASUREMENT SYSTEM 

In order to measure the performance of the auditee and the auditing 
organization, certain standard measurements are required. These 
measurements are formulated by utilizing the evaluation data obtained from 
the above procedure. 

5.6.1 Sum Evaluation Data 

When summing the evaluation data. Nonconformances, Verifications, 
Observations, and Commendations are accumulated separately. Count 
the total number of Nonconformances that had the Importance 
characteristic judged to be Critical and record as Nc. Perform a 
similar count for Mission (Nm), and Economic (Ne). Perform the same 
process for Verifications, Observations, and Commendations, 
respectively. These sums will be used during the calculation of the 
auditee metrics. Summations of the Impact characteristic are used for 
metrics of the auditing organization. 

Some organizations may find it desirable to place weighted values to 
each of the attributes in Diagram 3. Weighting the attributes is 
optional and may provide additional emphasis where deemed 
necessary. A specific method of weighting is beyond the scope of this 
document. 
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5.6.2 Assess Results 


Generally, measures are intended to provide information regarding the 
appropriateness of previous corrective actions. As such, interpretation 
requires a comparison of the present data to previous data. However, 
an analysis of the present data from this measurement system can 
yield immediate information regarding the health and status of the 
auditee. The valuations of Importance, Character, Frequency, and 
Impact are ordered in significance from top to bottom. An initial look 
at the valuations of the Nonconformances and Observations will 
identify the need for immediate concern, the degree of concern, and the 
potential for realization of the concern. 

As with any measurement system, accuracy will increase with more 
samples. Care must be taken to avoid judgments based on insufficient 
data. It may be necessary to perform further investigations. To avoid 
the anomalies associated with small sample sizes, the number of 
significant items in the audit plan should be greater than 35. Audits 
that have fewer than 35 must contain information necessary to justify 
the deviation. 

5.6.3 Auditee Metrics 

To facilitate continuous improvement, metrics are required to assess 
the auditee. The data for the metrics is derived from the summations 
of finding evaluations performed in paragraph 5.6.1. The metrics 
described below relate the Importance of survey and audit findings to 
the compliance of the auditee. When applying metrics to the results of 
a process audit, the auditee is the process. 

The values obtained from the metrics for any particular audit are not 
intended to indicate a good or bad level of performance. The utility of 
the metrics is in providing a value benchmark to determine if 
improvement has occurred. A comparison can be made between 
successive audits or between periods of time because the data is 
relational and may be summed over several audits. The data can also 
be subdivided to provide values for metrics that have a Systemic or 
Organizational character. 

A. Critical Compliance 

The purpose of this metric is to measure the auditee's 
effectiveness in complying with requirements having critical 
significance. If an audit does not include a review of critical 
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requirements, the critical effectiveness measure is "Not 
Applicable." This metric is calculated as follows: 

Ce = Cp/Fc = Cp/(Cp + Nc) 

where Ce is the Critical Effectiveness Measure, 

Fc IS THE NUMBER OF COMPLIANCE FINDINGS OR 
TOTAL FINDINGS LESS OBSERVATIONS, 

Cp is Critical Performance (the number of 

COMPLIANCE FINDINGS EXCLUDING 
NONCONFORMANCES IDENTIFIED AS CRITICAL), 

or Fc less Nc (Cp = Fc- Nc), and 

Nc IS THE NUMBER OF NONCONFORMANCES 
IDENTIFIED AS CRITICAL. 

Note: As Nc improves toward zero, Ce will approach one. 

B. Mission Compliance 

The purpose of this metric is to measure the auditee's 
effectiveness in complying with requirements having mission 
significance. If an audit does not include a review of mission 
requirements, the mission effectiveness measure is "Not 
Applicable." This metric is calculated as follows: 

Me = Mp/Fc = Mp/(Mp + Nm) 

WHERE Me IS THE MISSION EFFECTIVENESS MEASURE, 

Fc IS THE NUMBER OF COMPLIANCE FINDINGS OR 
TOTAL FINDINGS LESS OBSERVATIONS, 

Mp is Mission Performance (the number of 

COMPLIANCE FINDINGS EXCLUDING 
NONCONFORMANCES IDENTIFIED AS MISSION), 

or Fc less Nm (Mp = Fc - Nm), and 

Nm IS THE NUMBER OF NONCONFORMANCES 
IDENTIFIED AS MISSION. 

Note: As Nm improves toward zero, Me will approach one. 
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C. Economic Compliance 


The purpose of this metric is to measure the auditee's 
effectiveness in complying with requirements having economic 
significance. If an audit does not include a review of economic 
requirements, the economic effectiveness measure is "Not 
Applicable." This metric is calculated as follows: 

Ee = Ep/Fc = Ep/(Ep + Ne) 

where Ee is the Economic Effectiveness measure, 

Fc IS THE NUMBER OF COMPLIANCE FINDINGS OR 
TOTAL FINDINGS LESS OBSERVATIONS, 

Ep is Economic Performance (the number of 

COMPLIANCE FINDINGS EXCLUDING 
NONCONFORMANCES IDENTIFIED AS 

ECONOMIC), OR FCLESS Ne (EP = FC- Ne), 
AND 

NE IS THE NUMBER OF NONCONFORMANCES 
IDENTIFIED AS ECONOMIC. 

Note: As Ne improves tow ard zero, Ee will approa ch one. 

D. Overall Compliance 

The purpose of this metric is to measure the auditee's overall 
compliance. This metric is calculated as follows: 

Oc = Tvc/Fc = Tvc/(Tvc + Nt) 

WHERE Oc IS THE OVERALL COMPLIANCE OF THE AUDITEE, 

FC IS THE NUMBER OF COMPLIANCE FINDINGS OR 
TOTAL FINDINGS LESS OBSERVATIONS, 

Tvc is the Total number of Verifications and 
Commendations, and 

Nt is the Total number of Nonconformances. 
Note: As Nt improves toward zero, Oc will approach one. 
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The auditee metrics provide management with an indication of 
the level of compliance pertaining to critical responsibilities, 
mission -related activities, and economy of operations. These 
metrics, along with overall compliance, will be used to 
determine if the systems and processes being audited are 
benefiting from the survey or audit process. A decrease in the 
mission or economic compliance metric from one audit to the 
next will alert management to the potential need for action 
through tr ainin g, increased discipline, additional special audits, 
or other emphasis methods, as appropriate. A decrease in the 
critical or overall compliance metric from one audit to the next 
or an exceptionally low value during an initial bench marking 
audit will indicate the need for management attention at the 
highest level. An increase in overall compliance in the absence 
of any specific critical or mission-related problems may allow 
the auditing organization to justify less frequent audits, thereby 
maximizing the utilization of resources. 

6.6.4 Trend 

Survey and audit metrics shall be trended to exhibit and 
demonstrate any tendencies revealed by successive evaluations 
of similar characteristics. The results of this analysis may 
require further investigation into selected areas. 

6.6.5 Report 

The auditor should prepare reports in a concise manner. 
Recommendations for corrective action and improvement must 
be explicit. It is advisable to justify recommendations by 
indicating appropriate metrics and identifying the results of 
applicable discussions with management. Judgments may be 
substantiated with trend data. 

5.6.6 Auditing Organization Metrics 

To facilitate continuous improvement, metrics are required to 
assess the survey and audit organization effectiveness. The 
data for the metrics is derived from the summations of finding 
evaluations performed in paragraph 5.6.1. The survey and 
audit organization's effectiveness in achieving the desired 
impact is related to the success in making recommendations 
that are acceptable to the auditee. It will be necessary to review 
the auditee's corrective action plan to determine implemented 
recommendations. 
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A. Safety 

The purpose of the safety metric is to measure the auditing 
organization’s effectiveness in influencing areas that impact 
safety. The safety metric is calculated as follows: 

SE = OSA/(OSA + OSN) 

WHERE SE IS THE EFFECTIVENESS MEASURE OF THE 

AUDITING ORGANIZATION INVOLVING SAFETY, 

OSA IS THE NUMBER OF OBSERVATIONS IDENTIFIED 

AS AN IMPACT TO SAFETY WHERE THE AUDITOR 
RECOMMENDATION WAS ACCEPTED BY THE 
AUD1TEE, AND 

OSN IS THE NUMBER OF OBSERVATIONS IDENTIFIED 

AS AN IMPACT TO SAFETY WHERE THE AUDITOR 
RECOMMENDATION W'AS NOT ACCEPTED BY 
THE AUDITEE. 

Note: As Osn improves tow,\rd zero, Se will approach one. 

B. First Time Quality 

The purpose of the first time quality metric is to measure the 
auditing organization's effectiveness in influencing the 
occurrence of first time quality. The metric is calculated as 
follows: 

FTQe = Ofa/(Ofa + Ofn) 

where FTQe is the Effectiveness measure of the 

AUDITING ORGANIZATION INVOLVING FIRST 

Time Quality, 

Ofa is the number of Observations identified 

AS AN IMPACT TO FIRST TIME QUALITY WHERE 
THE AUDITOR RECOMMENDATION WAS 
ACCEPTEE ' BY THE AUDITEE, AND 

Ofn IS THE NUMBER OF OBSERVATIONS IDENTIFIED 
AS AN IMPACT TO FIRST TIME QUALITY WHERE 
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THE AUDITOR RECOMMENDATION WAS NOT 

Accepted by the auditee. 

Note: As Ofn improves toward zero, FTQe will approach 

ONE. 

C. Product Quality 

The purpose of the product quality metric is to measure the 
auditing organization's effectiveness in influencing improved 
product quality. The metric is calculated as follows: 

PQe = Opa/(Opa + Opn) 

where PQe is the Effectiveness measure of the 

AUDITING ORGANIZATION INVOLVING PRODUCT 

Quality, 

Opa is the number of Observations identified 

AS AN IMPACT TO PRODUCT QUALITY WHERE 
THE AUDITOR RECOMMENDATION WAS 

Accepted by the auditee, and 
Opn is the number of Observations identified 

AS AN IMPACT TO PRODUCT QUALITY WHERE 
THE AUDITOR RECOMMENDATION WAS NOT 

Accepted by the auditee. 

Note: As Opn improves toward zero, PQe will approach 

one. 

D. Overall Effectiveness 

The purpose of this metric is to measure the auditing 
organization’s overall effectiveness. The metric is calculated as 
follows: 

Oe = Ooa/(Ooa + Oon) 

where Oe is the Overall Effectiveness measure of 

THE AUDITING ORGANIZATION, 

OOA IS THE TOTAL NUMBER OF OBSERVATIONS 

W HERE THE AUDITOR RECOMMENDATION WAS 

Accepted by the auditee, and 
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OON IS THE TOTAL NUMBER OF OBSERVATIONS 

WHERE THE AUDITOR RECOMMENDATION WAS 

Not Accepted by the auditee. 

Note: As Oon improves toward zero, Oe will approach one. 

The auditing organization metrics provide management with an 
indication of the effectiveness of personnel in adding value to 
systems through audit, especially in the areas of safety, first 
time quality and overall product quality. The metrics should be 
accumulated for a period of time, i.e., calendar year, fiscal year, 
or performance period. Trends that exhibit deviations from a 
measure of one alert management to the need for additional 
training or increased emphasis in these areas. The Overall 
Effectiveness metric includes all of the above and those 
Observations impacting cost. Trends that exhibit deviations 
from a measure of one indicate that auditors are not 
communicating properly will all levels of cognizant personnel 
when making recommendations for improvement. This may be 
caused by inadequate auditor training or insufficient duration of 
the audit. 

5.6.7 Corrective Action Metrics 

Auditees are required to submit a Corrective Action Plan in 
response to the survey and audit report. The plan identifies 
corrective actions and improvements that the auditee intends to 
perform. Each commitment requires an estimated completion 
date. The auditing organization is responsible for evaluating 
and concurring in the Corrective Action Plan and any 
subsequent revisions. 

A. Corrective Action Effectiveness 

The purpose of the corrective action effectiveness metric is 
to measure the auditee's effectiveness in accomplishing 
planned corrective actions and improvements. The 
corrective action effectiveness metric is calculated as 
follows: 

Re = Completed/Planned 

where Re is the Corrective Action 

Effectiveness, "Completed" is the 
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NUMBER OF COMMITMENTS ACHIEVED, 

and "Planned" is the number of 

COMMITMENTS SCHEDULED. 

Corrective Action Effectiveness (Re) can be calculated at 
major milestones or at the estimated completion of the 
plan. 

B. Corrective Action Timeliness 

The purpose of the corrective action timeliness metric is 
to measure corrective actions as originally scheduled and 
identified in the auditee's corrective action plan. 

Rt = Completed On Schedule/Planned 

where Rt is the Corrective Action Timeliness, 
"Completed on schedule" is the 

NUMBER OF COMMITMENTS ACHIEVED 
ON TIME, AND "PLANNED" IS THE 
NUMBER OF COMMITMENTS SCHEDULED. 

The Corrective Action Effectiveness metric provides 
management with an indication of completed work. Until 
this metric equals one, resources are required to complete 
proposed actions. If the metric does not indicate a 
continuous approach toward one, the auditing 
organization will be alerted to a lack of progress and may 
fin d it necessary to bring the condition to the attention of 
the auditee’s management. The Corrective Action 
Timeliness metric provides management with an 
indication of how adequately the resources are meeting 
commitments. If a growing difference is noted between 
Re and Rt, the auditee will be alerted to the development 
of scheduling problems. 

5.7 DATA REQUIREMENTS 

Each organization shall develop their own procedure on how to integrate the 
collection of required data. Data collection shall be consistent with the 
requirement to provide periodic computation of measures and reporting. All 
data that is collected as part of the required indicators shall be available for 
both in-house and NASA audit or review. 


37 




KSC SURVEY AND AUDIT PROGRAM 
SECTION VI 


NASA RESPONSIBILITIES 

6.1 GENERAL 

The responsibilities for assessing conformance to SRM&Q and SRM&Q 
Assurance (SRM&QA) requirements are shared between the Primary 
Organizations, Project Managers, and the SR&QA Organization. In general, 
Primary Organizations are responsible for SRM&Q within their 
organization, as well as, providing for SRM&Q and SRM&QA within 
contractor organizations under their management. In addition, Primary 
Organizations, Project Managers are responsible for self-assessment 
activities and supporting the SR&QA Organization, which includes 
fur nishin g auditors, or subject matter experts as team members to 
participate in SRM&Q and SRM&QA independent assessments. The 
SR&QA Organization is responsible for providing SRM&Q and SRM&QA 
support to Primary Organizations, Project Managers, oversight of self- 
assessment activities, as well as, providing leadership and integration of 
independent assessments of SRM&Q and SRM&QA implementation 
effectiveness through survey and audit activities. 

6.2 PRIMARY ORGANIZATIONS AND PROJECT MANAGERS 
Primary Organizations and Project Managers are responsible for: 

A. Performing activities within their organization and within contractor 
organizations under their management in accordance with safety, 
reliability, maintainability and quality requirements. 

B. Providing resources for contractor SRM&QA activities, including the 
training necessary to implement a process oriented survey and audit 
program. 

C. Designating the SR&QA Organization as the SRM&QA management 
representative for contracts under their management. 

D. Supporting the SR&QA Organization survey and audit function with 
subject matter experts trained and certified in accordance with the 
requirements of this plan. 
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E. Performing SRM&Q self-assessments to determine conformance to 
SRM&Q requirements and the attainment of first time quality in these 
disciplines. 

F. Supporting SRM&QA surveys and audits of their organization and 
contractor organizations under their management. 

G. Concurring with the findings of independent assessments, preparing 
corrective action plans, and implementing corrective actions. 

H. Reporting status of corrective actions significant enough to warrant 
tracking by the SR&QA Organization. 

I. Reporting to the Center Director and to the Director of Safety, 
Reliability and Quality Assurance areas of SRM&QA nonconformance, 
which their organization need not be required to meet, along with the 
rationale for departure from the requirement. 

6.3 SR&QA ORGANIZATION 

The SR&QA Organization is responsible for: 

A. Establishing and interpreting SRM&Q policy, requirements, assurance 
activities, and supporting Primary Organizations and Project 
Managers in implementing SRM&Q within their organization and 
within contractor organizations under their management. 

B. Establishing and implementing an SRM&QA independent assessment 
of NASA-KSC peer organizations and their contractor SRM&Q 
activities through inspection, surveillance, and surveys and audits of 
processes and products. 

C. Performing SRM&QA self-assessments to determine conformance to 
SRM&QA requirements and the attainment of first time quality in 
these disciplines. 

D. Providing integration and leadership for the survey and audit function 
through a cadre of experts (Senior Auditors) in management and 
operation of the survey and audit function. This cadre will also 
provide oversight of self-assessments of NASA-KSC and contractors. 

E. Supporting the Survey and Audit organization with SRM&Q subject 
matter experts who are trained and certified in accordance with the 
requirements of this plan. These subject matter experts (Auditors) will 
participate in surveys and audits to augment Senior Auditors. During 
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the course of an audit, the auditor is dedicated full time to the 
performance of the survey and audit function. 

F. Provide interface and liaison (representing KSC) for problem 

resolution and requirement determinations with peer Centers, NASA 
Program Managers, and NASA Headquarters in the area of SRM&Q 
and SRM&QA surveys and audits. 

6.3.1 SRM&Q Engineering Organizations 

The SRM&Q Engineering Organizations are responsible for: 

A. Performing SRM&Q self-assessments to determine conformance 
to SRM&Q requirements and the attainment of first time 
quality in these disciplines. 

B. Supporting the Survey and Audit Organization with SRM&Q 
subject matter experts trained and certified in accordance with 
the requirements of this plan. These subject matter experts 
(Auditors) will participate in surveys and audits to augment 
Senior Auditors and are responsible for providing 
determinations of the significance and value added of findings 
within the area of their expertise. During the course of an 
audit, the auditor is dedicated full time to the performance of 
the survey and audit function. Departures from this 
requirement will be at the discretion of the Senior Auditor. 

C. Reporting to the Director of Safety, Reliability and Quality 
Assurance areas of SRM&QA nonconformance, which their 
organization need not be required to meet, along with rationale 
for departure from the requirement. 

D. Determining corrective actions significant enough to warrant 
statusing by the survey and audit function. 

E. Evaluating the performance of the survey and audit function. 

6.3.2 Quality Assurance Organization 

The Quality Assurance Organization is responsible for: 

A. Perfor min g SRM&QA self-assessments to determine 

conformance to SRM&QA requirements and the attainment of 
first time quality in these disciplines. 
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B. Supporting the Survey and Audit organization with Quality 
Assurance subject matter experts who are trained and certified 
in accordance with the requirements of this plan. These subject 
matter experts (Auditors) will participate in surveys and audits 
to augment the Senior Auditors. The auditors are responsible 
for providing requirement adequacy determinations and the 
value added of findings within the area of their expertise. 

During the course of an audit, the Auditor is dedicated full time 
to the performance of the survey and audit function. 

Departures from this requirement will be at the discretion of the 
Senior Auditor. 

C. Concurring with the findings of independent assessments, 
preparing corrective action plans, and implementing corrective 
actions. 

D. Reporting to the Director of Safety, Reliability and Quality 
Assurance those areas of Quality Assurance nonconformance, 
which the organization need not be required to meet, along with 
rationale for departure from the requirement. 

E. Evaluating the performance of the survey and audit function. 

F. Determining corrective actions significant enough to warrant 
statusing by the survey and audit function. 

6.3.3 Survey and Audit Organization 

The Survey and Audit Organization is responsible for: 

A. Providing oversight of NASA-KSC and contractor self- 
assessment activities. 

B. Providing leadership and integration of surveys and audits of 
NASA-KSC organizations, processes, and products in the area of 
SRM&Q implementation. 

C. Providing leadership and integration of surveys and audits of 
contractor project management, engineering, and operational 
organizations in the implementation of SRM&Q requirements. 

D. Providing leadership and integration of surveys and audits of 
contractor SRM&QA capability, implementation, and 
effectiveness. 
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E. Escalating residual issues for resolution. 

F. Statusing corrective actions significant enough to warrant such 
action. 

G. Consolidating schedules and completions of KSC surveys and 
audits. 
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KSC SURVEY AND AUDIT PROGRAM 
SECTION VII 


PROGRAM REPORTS 

7.1 GENERAL 

To facilitate continuous improvement, periodic reports shall be required by 
both NASA and contractor organizations performing surveys and audits. 

7.2 PROGRAM REPORTS 

In conjunction with unifying the Kennedy Space Center’s survey and audit 
community, each contractor shall make the following inputs to the NASA 
survey and audit organization: 

A. Schedule/ Activity: All activity planned for the next 6 months and 
actual activity for the previous 3 months shall be reported quarterly 
(NASA fiscal year). 

B. Reports: Copies of all survey and audit reports. 

C. Measures: Survey and Audit effectiveness measurements and trends, 
identified in Section V, shall be reported quarterly (NASA fiscal year). 

D. Summaries: Survey and Audit summaries shall be submitted semi- 
annually unless contractually required otherwise. 
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KSC SURVEY AND AUDIT PROGRAM 
APPENDIX A 

SURVEY AND AUDIT PLAN GUIDE 


1.0 INTRODUCTION 

1.1 PURPOSE 

1.2 SCOPE 

1.2.1 On-Site Elements/Subordinate Organizations 

1.2.2 Off-Site Vendors/Suppliers 

1.3 APPLICABLE DOCUMENTS 

1.4 DEFINITIONS 

2.0 ORGANIZATIONAL RESPONSIBILITIES 

2.1 GENERAL 

2.2 SURVEY AND AUDIT ORGANIZATION 

2.3 SRM&QA ORGANIZATIONS 

2.4 MANAGEMENT 

3.0 APPROACH 

3.1 CONCEPT 

3.2 REQUIREMENT 

The survey and audit program is a program focused primarily on the 
evaluation of the conformance and effectiveness of processes. 

3.2.1 What is Expected 

Implementation of a process oriented survey and audit program 
which establishes a closed loop system for continual 
improvement. Judgments of the significance of compliance 
findings shall be made by the survey and audit teams subject 
matter experts before the findings are formalized. 
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3.2.2 Value Added 

(Only value added findings will be documented. The progr am 
will measure its performance through increases in first time 
quality of process improvements initiated as a result of 
recommendations.) 

3.2.3 Principles 

(This program is based on the principle that surveys and audits 
exist to provide an independent assessment of conformance, 
worth of conformance, and identification of improvement 
opportunities.) 

3.2.4 Implementation 

(Implementation shall be described through procedures. The 
plan shall include internal and external survey and audit 
functions covering design, engineering, procurement, as well as, 
operation and maintenance activities.) 

3.3 ON-SITE ACTIVITIES 

(Describe the environment of the on-site process and the flexibility 
permitted because of that environment. Explain the application of 
survey techniques to both processes and organizations, as well as, 
differences in expected outcomes from each general category.) 

3.3.1 Surveys (Place Different Kinds Here) 

(The discussion should focus on the general or generic flow of a 
survey keeping in mind the intent of a survey is to determine 
capability to perform through observations, judgments, and 
objective evidence. Explain the processes and decision points 
identified as to purpose, data requirements, participants, 
decisions, and decision makers.) 

3.3.2 Audits (Place Different Kinds Here) 

(The discussion should focus on the general or generic flow of a 
audit keeping in mind the intent of an audit is to determine 
compliance, worth of requirements, and improvement 
opportunities through determinations based on objective 
evidence.) 
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3.4 OFF-SITE ACTIVITIES 


3.4.1 Surveys (Place Different Kinds Here) 

(The discussion should focus on the general or generic flow of a 
survey. Explain the processes and decision points identified as 
to purpose, data requirements, participants, decisions, and 
decision makers.) 

3.4.2 Audits (Place Different Kinds Here) 

(The discussion should focus on the general or generic flow of an 
audit keeping in mind the intent of an audit is to determine 
compliance, worth of requirements, and improvement 
opportunities through determinations based on objective 
evidence.) 

4.0 RESOURCE IMPACTS 

4.1 AUDITOR QUALIFICATIONS AND RESPONSIBILITIES 

4.1.1 Lead or Chair Auditor (Senior Auditor Designated 
to Lead The Survey or Audit Activity) 

4.1.2 Senior Auditor (Meihber of Audit Organization 
with Highest Skill Level in Audit Technology) 

4.2 TRAINING AND CERTIFICATION 

6.0 MEASUREMENTS AND TRENDS 

6.1 SURVEYS (For both the activities surveyed and Survey Team Activity 
effectiveness and efficiency) 

6.1.1 Data Collection 

6.1.2 Data Processing 

6.1.3 Indicators 

6.2 AUDITS (For both activities surveyed and Audit Team Activity 
effectiveness and efficiency) 

6.2.1 Data Collection 

6.2.2 Data Processing 

5.2.3 Indicators 
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6.0 PROGRAM REPORTS 
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